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© Multimedia telemeeting terminal device, communication system and manipulation method thereof. 



© In a telemeeting terminal for performing commu- 
nication between remote locations (201) by using 
multiple media, the other party of a meeting is 
identified in a display picture and thereafter switch- 



ing to a picture for presenting and explaining materi- 
als (i.e., a principal picture of the meeting) is per- 
formed. As a result, it is possible to keep secret 
materials used in the meeting. 



F I G. 2 



(M 

< 

00 

w 



Q. 



20! 



COMMUNI- 
CATION 
TERMINAL 



202 



NETWORK 



20! 





COMMUNI- 
CATION 
TERMINAL 








203 


20I 




ELECTRONIC 

SWITCHING 

SYSTEM 




COMMUNI- 
CATION 
TERMINAL 


I — 





Xerox Copy Centre 



1 



EP 0 410 378 A2 



2 



MULTIMEDIA TELEMEETING TERMINAL DEVICE, COMMUNICATION SYSTEM AND MANIPULATION METH- 
OD THEREOF 



BACKGROUND OF THE INVENTION 

The present invention relates to telemeetlng 
terminal devices for performing communication be- 
tween remote locations by using multiple media 
(i.e., multimedia telemeeting terminal devices), and 
in particular to a screen display method for keeping 
materials of previous arrangements secret. 

Conventional terminal devices are described in 
JP-A-61 -114641. In terminals allowing communica- 
tion of audio data and image or code data, audio 
system service messages, data system messages 
or communication data are automatically displayed 
in varied picture configuration in accordance with 
the use situation of the user. 

In case one terminal device (hereafter simply 
referred to as terminal) is used in common by a 
plurality of users, switching to pictures (explanation 
pictures) used for presenting materials mutually to 
give explanations during a meeting is immediately 
performed. The above described prior art thus has 
a problem in keeping a secret. 

SUMMARY OF THE INVENTION 

An object of the present invention is to realize 
secret keeping by providing a step for necessarily 
identifying the other party of the meeting which is 
subject of the terminal before picture switching is 
performed during a meeting. 

According to one aspect a user is provided 
with a picture for identifying the other party of the 
meeting so that the user may easily identify the 
other party of the meeting. 

According to a further aspect convenient termi- 
nal use suitable for the situation of the meeting is 
provided by automatically providing multiwindow 
display of multiple media centering around a re- 
quired principal medium according to the progress 
of the meeting. Thereby troublesome manipulation 
of the terminal user such as window setting and 
window arrangement are avoided and hence fixed 
windows for multiwindow display of the provided 
multiple media are adopted. 

The other party of the meeting may be iden- 
tified when the communication line has been con- 
nected by automatically displaying a picture image 
before the meeting. 

A closer resemblance to an interview meeting 
should be achieved by providing a picture before 
the meeting centering around the dynamic picture 
display of the person of the other party in addition 
to the principal picture of the meeting having media 



other than dynamic pictures. 

Simple means should be provided for informing 
the terminal of identification of the other party by 
providing menu selection means as means for in- 
5 forming the terminal that the other party has been 
identified. 

The principal pictures of the meeting may be 
displayed under the condition that the screen is 
divided into an image display area common to the 
70 terminal of the other party and a menu display area 
of meeting support for controlling the common im- 
age. 

Selection of the menu for identifying the other 
party can be facilitated by displaying a menu 

75 (hereafter used to mean a selected menu item or a 
picture or a display frame displaying a menu item) 
for informing the terminal that the other party has 
been identified on the common image display area 
Independently of the meeting support menu dis- 

20 played on the menu display area. 

An image, which is input at the terminal of the 
other party can be displayed at its own terminal in 
pictures before meeting for identifying both sides 
before the start of the meeting and displaying 

25 either of images inputted at its own terminal and 
images inputted at the terminal of the other party 
on both terminals in principal pictures of the meet- 
ing. 

In order to achieve the above described princi- 
30 pal object, there are provided means for identifying 
the other party of the meeting, means for informing 
the terminal device of the other party that the other 
party of the meeting has been identified, means for 
storing principal pictures of the meeting, and 
35 means for displaying the principal pictures thus 
stored. 

For the solution of the further aspects of the 
invention there is provided one of a combination of: 

- means for storing a picture before meeting; 

40 - means for displaying multiple media with multiple 
windows and means for storing initial display posi- 
tions and sizes of respective media; 

- means for storing fixed multiwindow display of 
multiple media according to the progress of the 

45 meeting; 

- means for automatically displaying the picture 
before meeting when communication lines have 
been connected; 

- means for audio communication, means for dis- 
.50 playing at least a dynamic picture in the picture 

before meeting and means for displaying media 
other than the dynamic picture in the principal 
picture of the meeting; 

- means for storing menu pictures, means for dis- 
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playing menu pictures, means for making menu 
selection possible and means for informing the 
other party of the selected menu; 

- means for storing a menu display area controlling 
common images and means for storing contents of 5 
the displayed menu; 

- means storing menus for informing the terminal 
that the other party has been identified and means 
for combining the above described menus and 
displaying the result on the common image display io 
area; and/or 

- means for transmitting images inputted at its own 
terminal to the terminal of the other party at the 
stage of identification of both sides before meeting, 
means for displaying images transmitted from the rs 
terminal of the other party at that stage, means for 
selecting either images inputted at its own terminal 

or images inputted at the terminal of the other 
party in the principal picture of the meeting, and 
means for displaying images inputted at its own 20 
terminal in case input images should be displayed 
on its own terminal. 

Thereby following advantageous effects are 
achieved: 

® The other party of a meeting is identified by 25 
the means for identifying the other party of the 
meeting. The terminal of the other party is in- 
formed that the other party of the meeting has 
been identified by the means for informing the 
terminal of the other party that the other party of 30 
the meeting has been identified. Switching to 
principal picture of the meeting is performed 
simultaneously at both sides by using the 
means for storing the principal picture of the 
meeting and the means for displaying the princi- 35 
pal picture of the meeting. 
(D When the picture before the meeting are to 
be displayed, it is implemented by the means 
for storing the picture before meeting. 
® Multiple media are displayed with multiple 40 
windows according to a required scene by using 
the means for displaying multiple media with 
multiple windows and means for storing initially 
set display positions and sizes of respective 
media. *6 
® Multiple media are displayed with fixed 
multiple windows according to the progress of 
the meeting by storing the fixed multiwindow 
display configuration of multimedia information 
according to the progress of the meeting. so 
© When communication lines are connected, 
the picture before meeting is displayed simulta- 
neously on both screens by using the means for 
automatically displaying the picture before 
meeting when communication lines have been 55 
connected. 

© In the picture before meeting, it becomes 
possible to identify the other party by voices 



and images owing to possession of the means 
for displaying at least a dynamic picture image 
together with the means for audio communica- 
tion. Further, in the principal picture of the meet- 
ing, diverse presentation becomes possible by 
the possession of the means for displaying at 
least media other than dynamic picture image 
together with voices. 

© Owing to the means for storing menu pic- 
tures and the means for making menu selection 
possible, it becomes possible to select such a 
menu as to inform the terminal that the other 
party has been identified. Owing to the posses- 
sion of the means for informing the other party 
of the selected menu, switching to the principal 
picture of the meeting is performed in synchro- 
nism with the terminal of the other party. 
® Progress of the meeting is made possible by 
the means for storing the menu display area 
controlling common images and the means for 
storing the displayed menu. 
(§) Such a meeting environment as to facilitate 
manipulation is provided by storing menus for 
informing the terminal that the other party has 
been identified, combining these stored menus 
and displaying the result on the common image 
display area. 

© 

In the scene for identifying both parties before 
meeting, images inputted at its own terminal are 
transmitted to the terminal of the other party, 
and images transmitted from the terminal of the 
other party are displayed. In the principal picture 
of the meeting, either images inputted at its own 
terminal or images inputted at the terminal of 
the other party are displayed. In case input 
images are to be displayed at its own terminal, 
images are automatically switched and dis- 
played so as to meet the progress of the meet- 
ing. A convenient environment is thus provided. 

BRIEF DESCRIPTION OF THE DRAWING 

Figs. 1(A) to 1(B) exemplify displayed pictures 
according to an embodiment of the present in- 
vention; 

Fig. 2 exemplifies the entire configuration of a 
system according to an embodiment of 4he 
present invention; 

Fig. 3 is an exterior view of a terminal according 
to an embodiment of the present invention; 
Fig. 4 is a block diagram showing hardware 
configuration of an embodiment of the present 
invention; 
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Fig. 5 shows an example of software configura- 
tion according to an embodiment of the present 
invention; 

Fig. 6 exemplifies display menus according to 
an embodiment of the present invention; 5 
Fig. 7 exemplifies storage of display menus 
according to an embodiment of the present in- 
vention; 

Rg. 8 exemplifies preservation of dispiay menus 
for respective pictures according to an embodi- io 
ment of the present invention; 
Rg. 9 exemplifies storage of menus to be dis- 
played according to an embodiment of the 
present invention; 

Rg. 10 exemplifies contents of processing per- 75 
formed when menus are actually selected ac- 
cording to an embodiment of the present inven- 
tion; 

Fig. 11 exemplifies data inputted from a stylus 
pen according to an embodiment of the present 20 
invention; 

Rg. 12 is an exterior view of a stylus pen 
according to an embodiment of the present in- 
vention; 

Rg. 13 exemplifies processing of data inputted 25 
via a stylus pen according to an embodiment of 
the present invention; 

Fig. 14 exemplifies a still picture processing 
apparatus according to an embodiment of the 
present invention; 30 
Rg. 15 exemplifies command responses of a 
main body and the still picture processing ap- 
paratus according to an embodiment of the 
present invention; 

Fig. 16 exemplifies a FAX document processing 35 
apparatus according to an embodiment of the 
present invention; 

Fig. 17 exemplifies an input/output control image 
processing section shown in Rg. 16; 
Fig. 18 exemplifies a communication sequence 40 
according to an embodiment of the present in- 
vention; 

Fig. 19 is a flow chart of picture switching caus- 
ed at the transmitting side by menu selection 
according to an embodiment the present inven- 45 
tion; 

Rg. 20 is a flow chart of picture switching caus- 
ed at the receiving side by menu selection ac- 
cording to an embodiment of the present inven- 
tion; 50 
Figs. 21 to 23 exemplify pictures displayed ac- 
cording to the present invention in case meeting 
is held between two locations or more; 
Fig. 24 exemplifies conservation of the minutes 
at the time of picture switching; and 55 
Fig. 25 shows an embodiment of means for 
identifying the other party of a meeting. 



DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

An embodiment of the present invention will 
now be described in detail. 

Fig. 2 shows an example of a system accord- 
ing to the present invention. Numeral 201 denotes 
a communication terminal device, and numeral 202 
denotes a communication network whereas numer- 
al 203 denotes an electronic exchange. Users of 
communication terminal devices 201 are connected 
to various communication media. For example, 
communication terminal devices 201 are connected 
to the electronic exchange 203 installed in prem- 
ises of an office building or a public network exist- 
ing between premises such as a private line net- 
work or an ISDN network to hold a communication 
meeting among remote locations. 

Fig. 3 is an exterior view of a terminal device 
according to the present invention. A general pur- 
pose personal computer system is used as a basic 
component, and hardware for implementing the 
function of the present invention is added to it. 
Numeral 301 denotes a main body of the persona! 
computer, 302 a CRT dispiay device, 303 a trans- 
parent tablet, 304 a television camera, 305 a key- 
board, 306 a scanner, 307 a printer, 308 a handset 
having a microphone and a loudspeaker, 309 an 
additional hardware, 310 a plug receptacle for con- 
nection to an ISDN network, and 311 a stylus pen. 
The personal computer main body 301 is a princi- 
pal processing section for implementing a commu- 
nication meeting and in charge of operation and 
administration of the terminal device. The CRT 302 
displays full-color still picture documents and dy- 
namic pictures (represented by 8 bits for each of 
R, G and B and) inputted from the television cam- 
era 304 and FAX documents inputted from the 
scanner 306 as well as menus for advancing the 
meeting. A menu to be selected is directly selected 
out of displayed menus by using the stylus pen 
311 on the transparent tablet added to the surface 
of the CRT 302. The keyboard 305 is a device for 
inputting a document name or the like for admin- 
istration into a document inputted from the scanner 
306 or the television camera 304. Data inputted 
from the keyboard 305 are converted into Japa- 
nese by using Japanese syllabary - Chinese char- 
acters conversion function prepared in the personal 
computer main body 301 . The printer 307 is a 
device for outputting picture data obtained by 
superposing a drawing inputted via the stylus pen 
31 1 during a meeting on a FAX document inputted 
via the scanner 306. The handset 308 functions to 
input and output voices during the meeting. The 
additional hardware 309 is a hardware used exclu- 
sively for substitutingly executing information pro- 
cessing such as processing of full-color still picture 
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documents and FAX documents that the personal 
computer main body 301 alone cannot afford to 
process. The ISDN plug receptacle 310 is a con- 
nection receptacle for connection to a public net- 
work such as an ISDN network. This implements 
Interface (hereafter abbreviated to l/F) based upon 
or nearly based upon standards with respect to a 
public network. 

Fig. 4 is a block diagram showing the entire 
configuration of a terminal device according to the 
present invention and Is roughly divided into the 
following two sections. © Numeral 410 denotes a 
personal computer section which is a main body of 
the meeting terminal device. ® Numeral 420 
denotes a meeting terminal adapter device which 
functions as an input/output section of the meeting 
terminal device main body. In the personal com- 
puter section 410, numeral 411 denotes a CPU 
comprising a personal computer, 412 an RS232C 
l/F, 413 a floppy disk, 414 a hard disk, 415 a 
display control section, 416 a display information 
storage section, and 417 an extension bus l/F for 
connection to a bus of the meeting terminal adapt- 
er 420. In the meeting terminal adapter 420, nu- 
meral 421 denotes a peripheral controller, 422 a 
voice CODEC, 423 a display composition section, 
424 an image CODEC, 425 a communication con- 
trol section, and 430 an ISDN line. In the terminal 
device as a whole, a multiprocessor scheme is 
adopted by using the CPU 411 included in the 
meeting terminal device main body 410 as a princi- 
pal part thereof. Respective sections included in 
the meeting terminal adapter device 420 are so 
connected as to undergo input/output by the CPU 
411 and execute independent processing in par- 
allel. Function of the personal computer section 
410 and the meeting terminal adapter 420 will now 
be described. First of all, in the personal computer 
section, the floppy disk 413 for preserving docu- 
ments is used to preserve documents which have 
been used in a meeting together with administra- 
tion-information in operating the meeting such as 
the date and hour fixed for the meeting and names 
of participants when the communication meeting 
has been finished. When meeting materials are 
produced before the meeting and when additional 
materials are transferred during the meeting, the 
floppy disk 413 is used as a document Input 
source together with documents newly inputted via 
the television camera 304 and the scanner 306. 
The hard disk 414 is used to store and administer 
not only documents used in the meeting but also 
information supplied from the terminal user and the 
other party of the meeting. The display device CRT 
302 of the personal computer in the present em- 
bodiment has not capability to deal with fulhcolor 
data having 8 bits in each of R (red), G (green), 
and B (blue). In the display control section 415, 



therefore, FAX documents comprising binary inn- 
ages, pointing to its own drawing Inputted from the 
transparent tablet 303, pointing to a drawing trans- 
mitted from the communication control section 425 

5 included in the meeting terminal adapter 420, and 
menu pictures are combined. Full-color Images are 
generated in the Image CODEC 424 included in the 
meeting terminal adapter 420. On the other hand, 
image signals of the personal computer and full- 

w color image signals are combined in the display 
composition section 423 included in the meeting 
terminal adapter 420. As the display information 
storage section 416, conventional RAMs for work 
included in the personal computer are used. Dis- 

75 play picture information stored in the display in- 
formation storage section according to the progress 
of the meeting is a part of a software program 
providing an environment of operation of the com- 
munication meeting. When this software program 

20 rises, the display picture information is read out 
from the hard disk 414. Thereafter, the display 
picture information resides in the RAM for work 
until the above described meeting operation pro- 
gram is finished. The extension bus l/F 417 is a 

25 section for connecting the meeting terminal main 
body 410 to the meeting terminal adapter via an 
extension slot. The peripheral controller 421 is a 
section for processing FAX documents inputted 
from the scanner 306 and it. is included in external 

30 additional hardware. The peripheral controller 421 
performs control of the scanner 306 and the printer 
307, NMR coding for preservation and transmis- 
sion, NMR decoding for redisplaying preserved 
documents, and development of drawing at the 

35 time of printing. As configuration of interface be- 
tween the meeting terminal main body 410 and the 
peripheral controller 421, a multiprocessor scheme 
for implementing firmware in the peripheral control- 
ler 421 is adopted. Further, buses connecting be- 

40 tween processors are provided with independence 
and data are exchanged by communication be- 
tween processors so that those processors may 
operate independently. The voice CODEC 422 is a 
device for performing coding and decoding to ef- 

45 feet digital transmission of voices. The image 
CODEC 424 is a device for performing coding and 
decoding of full-color images, taking in images 
from the television camera, changing the position 
and size of images according to the progress of 

so the meeting, and writing data into a frame memory 
for full-color images included in the image CODEC. 
By using superimpose, the display composition 
section 423 combines the output signal of the 
display control section 415 included in the meeting 

55 terminal main body. 410 and the output signal of 
the image CODEC 424 and displays the result. In 
case N-ISDN is used as the line, for example, the 
communication control section 425 takes charge of 
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three lower layers of OSI (open systems interface). 

In order to enhance the wide use and flexibility 
of apparatus, the present embodiment has configu- 
ration based upon the personal computer. How- 
ever, an apparatus for exclusive use pursuing the 
performance and cost may also be used instead. 
Further, drawing and pointing coordinates are de- 
rived from the tablet and the stylus pen in the 
present embodiment However, any input device 
may be used so long as the Input device has 
function of drawing and pointing such as touch 
pannel, mouse, track ball, joystick, graphics tablet 
or light pen. The floppy disk 413 is used as a 
second storage device of a meeting file after the 
meeting has been finished. However, any device 
such as an optical disk device or a magnetic tape 
device may be used instead so long as the device 
has function of memory and storage. This also 
holds true for the disk 414 which is used as a first 
storage device of the meeting file from the time 
when the meeting materials are produced until the 
time when the communication meeting is finished. 
Further, the first storage device may be common to 
the second storage device. Further, function allot- 
ment to the display control section 415 and the 
display composition section 423 may be freely 
performed so far as respective processing capabil- 
ities permit. Further, the hardware may be single. 
The CRT 302 uses the single display scheme. 
From the viewpoint of reduction of hardware of the 
composition section, however, a plurality of dis- 
plays such as document display and dynamic pic- 
ture image display may be used instead. Although 
the display information storage section 416 com- 
prises a work RAM, it may comprise a ROM in 
case of an apparatus for exclusive use. 

Fig. 5 shows an example of software configura- 
tion of the entire telemeeting terminal. Numeral 551 
denotes a multitask OS, 552 a period task, 553 a 
menu analysis task, 554 a processing execution 
task, and 555 an interrupt task. 

In order to make great account of the man- 
machine interface, a period task having such visual 
characteristics as to provide men with feeling of 
natural input and having a raised priority level of 
processing is provided on the multitask OS 551. 
Coordinate data inputted from the transparent tab- 
let 303 are taken in at constant periods, and the 
menu analysis task 553 is started. In the menu 
analysis task 553, input data are analyzed. If a 
manu is selected, contents of indicated function 
meant by the menu is set, and the processing 
execution task 554 is started. In that task, process- 
ing according to the set value is executed. While 
consecutive processing such as drawing and point- 
ing is being executed, the processing execution 
task 554 is started as it is. In the processing 
execution task 554, the value set before continues 



as it Is, and in succession Its processing is per- 
formed. The interrupt task 555 is a task for execut- 
ing interrupt processing requested by the meeting 
terminal adapter 420. There are a processing end 

s notification issued from the peripheral controller 
421 , the image CODEC 424 and the display control 
section 415, and processing requests issued from 
the terminal of the other party via the communica- 
tion control section, such as a request to transfer 

10 processing data into the meeting terminal main 
body 410. 

Fig. 1 shows meeting progress pictures based 
upon the present invention. Fig. 1 (A), (B) and (C) 
show a picture 110 before meeting, a principal 

is picture 120 of the meeting, and a picture 130 after 
the meeting, respectively. In case one terminal is 
shared by a plurality of users, the principal picture 
120 of meeting is not immediately generated on 
the screen when lines are connected in order to 

20 keep a secret, but the principal picture 120 of the 
meeting is generated on the screen after the 
screen 110 before the meeting has been generated 
on the screen. With reference to Fig. 1(A), the 
picture 110 before meeting is divided into © a 

25 common image display area 111, © an area 112 
for displaying the other party dynamic picture im- 
age, © an area 113 for displaying its own dynamic 
picture image and @ a meeting progress menu 
display area 114. On the common image display 

30 area 111, a meeting start menu 115, a meeting 
suspension menu 116 and an area 117 for display- 
ing a dynamic picture image for identifying the 
other party are displayed. The dynamic picture 
image 119 for identifying the other party is dis- 

35 played on the display area 117. its own dynamic 
picture image 118 is displayed on its own dynamic 
picture image display area 113. 

With reference to Fig. 1(B), the common image 
display area 111 of the principal picture 120 of the 

40 meeting is illustrated as blank. By way of example, 
however, materials used in the meeting are dis- 
played on the common image display area 111. 
Further, a dynamic picture image 121 of the other 
party is displayed on the other party dynamic pic- 

45 ture image display area 112. 

With reference to Fig. 1(C), a communication 
end menu 135 and a meeting restart menu 136 are 
displayed on the common image display area 111 
of the picture 130 after meeting. 

so In a telemeeting terminal for performing com- 
munication between remote locations by using mul- 
tiple media, the other party of the meeting is iden- 
tified, and when switching to a picture for explain- 
ing presented materials (i.e., principal picture of the 

55 meeting) is performed. This switching method will 
hereafter be described in detail. 

For convenience, the picture 110 before meet- 
ing shown in Rg. 1(A), the principal picture 120 of 
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meeting shown in Fig. 1(B) and the picture 130 
after meeting shown in Fig. 1(C) are hereafter re- 
ferred to as picture 1, picture "2 and picture 3 t 
respectively. In the picture 1, the dynamic picture 
image 119 of the other party becomes a principal 
medium for identifying the other party. That is to 
say, the dynamic picture image of the other party 
is displayed large in the center of the screen to 
provide a picture suitable for identification of the 
other party. In order to check the image transmitted 
from that terminal device to the other party, its own 
dynamic picture image 118 may be subsidiarily 
displayed in the center of the right side of the 
screen. Further, the meeting start menu 115 and 
the meeting suspension menu 116 are disposed 
below the other party dynamic picture image dis- 
play area 117 shown in Fig. 1(A) to implement the 
function of smoothly starting the meeting and the 
function of suspending the meeting when the other 
party is absent. 

The recognition medium for identifying the oth- 
er party of the meeting is not limited to an image 
so long as it is a recognition medium peculiar to 
the other party. For example, the recognition me- 
dium is ID No., a fingerprint, a voiceprint, a retinal 
image, or a physical feature. If these are used, the 
operator need not watch the figure of the other 
party without fail to identify the other party, but the 
apparatus may automatically form a judgment. If it 
is especially necessary for the operator to finally 
identify the other party to keep a secret, double 
check comprising a combination of them may also 
be performed. 

Recognition of such a recognition medium can 
be implemented by means described below. A 
case where a fingerprint is used as a recognition 
medium will now be described by referring to Fig. 
25. 



© Inputting fingerprint 

For inputting a fingerprint, a scanner input 
switch 610 is first thrown to position b . A finger is 
pressed against a press sensor 600. The fingerprint 
is read by a scanner 306. Fingerprint read informa- 
tion (hereafter referred to as fingerprint information) 
is transmitted to a registration section 630 and 
stored therein. On the other hand, fingerprints of 
participants are thus registered one after another in 
respective terminal devices. It is a matter of course 
that fingerprint information thus registered is ex- 
changed via an ISDN line so that registration sec- 
tions of terminal devices of both parties of the 
meeting may have identical fingerprint information. 

® Receiving fingerprint information from the other 



party 

When the other party is to be identified, the 
scanner input switch 610 is thrown to position a 

s and a finger is pressed against the press sensor 60 
in the terminal device of the other party. A finger- 
print is thus read by the scanner 306 and transmit- 
ted to a connection bus 640. The fingerprint in- 
formation is received by the communication control 

70 section 425 of its own terminal device via an ISDN 
line. The fingerprint information thus received is 
temporarily stored into a collation section 620 via 
the connection bus 640. 

15 

® Collating fingerprint information of the other 
party 

The fingerprint information of the other party 

20 stored in the collation section 620 is compared and 
collated with fingerprint information of all partici- 
pants of the meeting registered in ® . Collation 
result information based upon the result is output- 
ted onto the connection bus 640. The CPU of the 

25 meeting terminal device main body 410 writes and 
stores this collation result information into the flop- 
py disk 413 and the hard disk 414. Administration 
software of the meeting terminal device main body 
takes over subsequent processing. 

30 The meeting progress menu area 114 will now 
be described in detail by referring to Fig. 6. In the 
picture 2, materials are presented and described, 
where facsimile documents and full-color still pic- 
ture documents become principal media. Such a 

35 medium is displayed on the common image dis- 
play area 111. In the picture state where materials 
are described, the other party dynamic picture im- 
age 121 and its own dynamic picture image 118 
are displayed on the upper right corner of the 

40 screen with a smaller picture size than that of the 
dynamic picture image displayed on the common 
image display area 111. This aims at enhancing 
the presentation effect by enhancing presence of 
meeting between remote locations and giving an 

45 explanation while observing the expression of the 
other party and response to a conversation. On the 
other hand, the state of meeting is finished by a 
meeting end menu 512 displayed on the right 
bottom corner. In the same way as the picture 1, 

60 the dynamic picture image 119 of the other party 
becomes a main medium in the picture 3 as well. 
The dynamic picture image 119 of the other party 
is displayed large in the center of the screen to 
provide a picture suitable for confirmation of the 

55 meeting end. At this time, its own dynamic picture 
image display is being transmitted to the other 
party. The communication end menu 135 and the 
meeting restart menu 136 disposed at the bottom 
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of the other party dynamic picture image 119 are 
selected when smooth end of communication is 
desired, when communication caused by a menu 
selection mistake is desired to be finished, or when 
contents of explanation are suddenly desired to be 
confirmed in the picture 130 after the meeting. Rg. 
6 shows details of the picture 2. Numeral 504 
denotes a token ownership display area, and its 
shaded area indicates possession of a token. The 
"token" means the right of priority of sending. The 
"token ownership" means the right of priority in 
picture switching. Numeral 505 denotes an ex- 
planation mode selection menu, 506 a document 
manipulation menu, 507 a meeting name display 
and document name selection menu, 508 a page 
name selection menu, 509 a print execution selec- 
tion menu, 510 a material transfer selection menu, 
511 a presented picture selection menu, and 512 
the meeting end selection menu. The presented 
material display area 501 is an area for displaying 
a FAX document, a full-color still picture document 
or the like as described before. The other party 
person dynamic picture image display area 502 
and its own person dynamic picture image display 
area 503 are areas for displaying person dynamic 
picture images. The token ownership display area 
504 indicates where the right of communication is 
present, and it is not a selection menu for acquiring 
the right of token. Transfer of the right of token is 
automatically performed by action of the user upon 
the CRT 302. In the following description, stress is 
laid upon menus for executing explanation. As a 
result of selecting a menu out of the menus dis- 
played on the meeting progress menu display area 
114, the displayed picture is switched. For exam- 
ple, the displayed document is switched from a 
FAX document to a full-color still picture document. 
The explanation mode selection menu 505 is used 
to select a description mode with respect to the 
displayed document and comprises © drawing for 
directly writing a memorandum on the document, 
@ drawing erasing for erasing a drawing contained 
in a small movable frame, and ® pointing for 
indicating a point of the document to be marked. 
The document manipulation menu 506 is a menu 
for changing the display area or the like without 
changing the displayed document. The document 
manipulation menu 506 comprises ® scroll for 
canging the display area, ® electronic memoran- 
dum paper for making a specified area of the 
displayed document white to facilitate description 
by means of drawing or the like, ® expansion 
display menu for displaying a specified area ex- 
panded with a fixed ratio as occasion demands, 
and 0 standard display menu for restoring the 
state before the expansion display. The present 
embodiment aims at making the number of times 
of manipulation performed until execution of menu 



processing small as far as possible. As for the 
expansion display, therefore, the area specification 
scheme with a fixed expansion ratio is adopted. As 
for the meeting name display and document name 

5 selection menu 507 and the page name selection 
menu 508, the storage structure of documents in 
the present embodiment is a pyramid structure 
comprising three stages, i.e. meeting-material- 
page. Although the menu performed at the present 

70 time is unique, there are a plurality of possibilities 
with respect to documents and pages. For specify- 
ing a displayed document (page), a document and 
a page must be specified. The meeting name dis- 
play and document name selection menu 507 is so 

15 configured to have four columns. The top column 
displays the meeting name. The second column 
displays the document name. As for the third col- 
umn, two left blocks indicate document alteration 
menus and the right block indicates display docu- 

20 ment No. / the number of documents. On the fourth 
column, participants of the meeting and the owner 
of the displayed document are displayed sepa- 
rately. By selecting a document alteration menu, 
the first page of a new document is displayed 

25 simultaneously at both terminals. The page name 
selection menu 508 changes over page display in 
the displayed document. Display is performed by 
taking four pages as the unit and a page menu to 
be displayed is selected. Two upper left menus 

30 changes over display by taking four pages as the 
unit. In the menu located below them, the largest 
page number of the displayed pages is displayed 
as compared with the total number of pages. 

The print selection menu 509 is a menu for 

35 printing out the document which is being displayed. 
The material transfer selection menu 510 is a menu 
for transferring a material which has suddenly be- 
come necessary during a meeting. In order to 
enhance the explanation capability in conformity 

40 with the above described document display, the 
presented picture selection menu 511 such as 
electronic blackboard and camera picture com- 
prises an electronic blackboard menu for clearing 
the whole of the common image display area 111 

45 (presented material display area) so that it may 
become white and explanations may be added 
there, and a camera picture menu for displaying 
dynamic picture images and directly displaying a 
solid body, a mock-up model and the like. The 

so principal picture of the meeting is finished by the 
meeting end selection menu 512. It is the user's 
intent to finish the meeting by selecting this menu. 
At that time, however, switching to the picture after 
meeting set by the terminal side as the progress 

55 scene of the meeting Is automatically performed. 

Succeedingly, switching of displayed picture 
performed by the terminal according to the 
progress of the meeting will now be described in 
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detail. First of all, as for the picture display method 
at the time when the display picture No. is defined, 
switching of the displayed picture in Fig. 1 will now 
be described. If menu selecting coordinates are 
inputted by a stylus pen, it is checked whether the 
inputted coordinates are within the range or not for 
each of menus of the displayed picture. At this 
time, there are two methods. In one method, menu 
contents are individually stored for each picture. In 
the other method, when there are menus common 
to pictures, each picture has bit information repre- 
senting the presence/absence of menus, with re- 
spect to the list of menus in order to reduce the 
amount of information. In the present embodiment, 
the latter method will be described. Figs. 7 to 10 
show the present invention. Fig. 8 shows an exam- 
ple of preservation of display menus for respective 
pictures. For each picture, displayed menus are 
preserved with respect to the list of selection 
menus. In the drawing, "1" represents presence of 
display whereas "0" represents absence of display. 
As shown in the drawing, selection menus are dealt 
with as numericals corresponding thereto through- 
out ail drawings. The display contents are stored 
separately by taking selection menu No. as the unit 
as shown in Fig. 7. It is now assumed that menu 
selection is performed by a stylus pen in the pic- 
ture 1 , for example. With reference to Fig. 9, selec- 
tion menus are successively searched beginning 
from the selection menu No. 1. If bit "1" is found, it 
is checked whether inputted coordinates are within 
the pertinent menu area or by referring to Fig. 8. If 
inputted coordinates are within the menu area, that 
menu area is selected. 

Fig. 10 shows an example of preservation of 
processing contents performed when menus are 
actually selected, if the selection menu No. 1 is 
selected in the picture 1, for example, switching to 
the screen 2 is performed. As for the display of 
switched pictures, a menu to be displayed is 
searched by referring to Fig. 9 and an area to be 
displayed is derived by referring to Fig, 8 whereas 
concrete display contents are derived by referring 
to Rg. 7. Display picture switching is thus per- 
formed according to the progress of meeting. 

As for the menu selection method, a stylus 
pen, for example, is used. 

Fig. 12 is an exterior view of a stylus pen. The 
stylus pen has a main switch 500 for defining 
actual inputted coordinates and two subswitches 
501 and 502 used to reflect the user's selection 
intent. Rg. 1 1 shows an example of data inputted 
via the stylus pen according to the present inven- 
tion. One coordinate includes five bytes. The first 
one byte is used to derive switch information of the 
stylus pen. Effective distance data represented by 
PX is used to judge whether the distance from the 
main switch 500 to the tablet surface is within a 



predetermined distance (3 to 5 mm) or not: The 
second to fifth bytes are binary number representa- 
tion of the X coordinate and the Y coordinate. In a 
scheme generally used as the menu selection 

5 method, menu selection becomes effective when 
either the subswitch 501 or 502 is pressed under 
the state that the main switch 500 is on. For 
making manipulation simpler, menu selection is 
herein attained by simply turning on the main 

10 switch 500. The following problems are posed at 
this time. (7) In case the user keeps the main 
switch on, a menu located in the same area of the 
switched picture is selected. © If an explanation 
such as drawing comes in the menu area of the 

75 picture 2 shown in Fig. 1, this menu is selected. 
Fig. 13 shows an example of solution to the above 
described problem according to the present inven- 
tion. On the ordinate, mode 0 and mode 1 are 
provided. Switching between the mode 0 and the 

20 mode 1 is performed by state values including 
values of the main switch SW1 and PX indicated 
on the abscissa. PX denotes information represent- 
ing whether the distance is within the effective area 
or not. SW1 denotes information representing 

25 ON/OFF of the main switch. "Within menu area" 
indicated for the states 4 and 5 indicates that the 
inputted coordinate value is within an area shown in 
Fig. 8 and an area specified by Fig. 9. The con- 
crete state switching method will hereafter be de- 

30 scribed. When the displayed picture is switched by 
menu selection, the mode is set at 0. Since a menu 
is selected by the input of the main switch at this t 
time, state 5 is established. When the stylus pen 
gets out of the effective distance in the state of 

35 mode 0, the mode is set at 1 as shown in Rg. 13. 
Menu selection is not allowed until this time. When 
the stylus pen is within the effective distance and 
the main switch is on in the mode 1, its inputted 
coordinates are judged. If the inputted coordinates 

40 are not within the menu area at this time, menu 
selection processing is not performed but the 
meeting operation program sets the mode at 0. If 
the inputted coordinates are within the menu area, 
menu processing including picture switching is per- 

45 formed and thereafter the mode is set at 0. Smooth 
menu selection is implemented by using the main 
switch alone as heretofore described. 

The description heretofore given relates to 
menu display. At the same time, one f s own dy- 

50 namic picture image and the dynamic picture im- 
age of the other party must be displayed in the 
picture 1 and the picture 3, whereas one's own 
dynamic picture image, the dynamic picture image 
of the other party and a document to be described 

55 must be displayed in the picture d. The display 
control method will hereafter be described. Rrst of 
all, full-color image display will now be described. 
A typical full-color image needs 8 bits for each of 
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R, G and B and hence needs 24 bits in total. In a 
typical personal computer, the processing capabil- 
ity and the display capability are insufficient. As 
described before with reference to Rg. 4, an exter- 
nal device of the personal computer performs pro- 
cessing and display, whereas the personal com- 
puter main body controls Its operation. Rg. 14 
shows an example of configuration of full-color 
image display hardware according to the present 
invention. Rg. 14 shows the image CODEC section 
in further detail. Numeral 1401 denotes an 
input/output buffer for the personal computer, 1402 
an input/output buffer for the communication con- 
trol section, 1403 a CPU for controlling the whole 
of the image CODEC, 1404 a coding/decoding de- 
vice for dynamic picture image and full-color still 
picture, 1405 an output frame memory, 1406 an 
input frame memory, and 1407 a section for con- 
verting an analog signal inputted from the TV cam- 
era into a digital signal. The display section is 
switched according to a request issued from the 
personal computer. The display control method of 
full-color images shown in Rg. 1 will hereafter be 
described. 

Rrst of all, the dynamic picture image of the 
other party person in the picture 1 will now be 
described. Dynamic picture image code data input- 
ted from the communication control section 425 are 
taken in via the input/output buffer 1402, decoded 
by the coding/decoding device 1404 for dynamic 
picture image and full-color still picture, and written 
into the output frame memory 1405. The display 
composition section 423 is so controlled by a dis- 
play changeover control signal supplied from the 
CPU 1403 that the written image signal may be 
combined with a personal computer signal and 
displayed. At the time of composition in displaying 
the dynamic picture image of the other party per- 
son, superposition of the dynamic picture image 
and the personal computer signal is not performed 
with due regard to the dynamic picture image, but 
only the dynamic picture image is displayed. Suc- 
ceedingly, the dynamic picture image of one's own 
person in the picture 1 is inputted from the TV 
camera 304. An NTSC signal, for example, under- 
goes analog-digital conversion in the A-D conver- 
sion section 1407 and is written into the input 
frame memory 1406. The image data written into 
the input frame memory 1406 is written into the 
output frame memory 1405 by the coding/decoding 
device for dynamic picture image and still picture. 
Display is performed in the same way as the 
dynamic picture image. The picture used for ex- 
plaining a material in the picture 2 will now be 
described. As for the person dynamic picture im- 
age, the input/output method is identical although 
the size and the display position of the dynamic 
picture image of the other party person are dif- 



ferent and processing of the dynamic picture im- 
age of the other party in the coding/decoding de- 
vice for dynamic picture image and still picture is 
slightly different. As for the processing of dynamic 

5 picture image of the other party in the picture 1 
and the picture 2, there are two methods as em- 
bodiments. In one method, the same processing is 
performed for input from the TV camera 304 and 
coding/decoding of the dynamic picture image and 

70 still picture in the coding/decoding device 1404 
whereas the size and position are changed de- 
pending upon whether the picture is picture 1 or 
picture 2 when the dynamic picture image is writ- 
ten into the output frame memory 1405. In the 

15 other method, the size is changed at the time of 
input and coding, and the display position is 
changed when the dynamic picture image is written 
into the output frame memory 1405. A case where 
a full-color still picture is displayed as a presented 

20 material of the picture 2 will now be described. 
Since the full-color still picture document is pre- 
served in the hard disk 414 of Fig. 4 as code data, 
it is first read out into the personal computer, and 
then taken into the image CODEC 424 via the 

25 extension bus l/F 417 and the input/output buffer 
1401. In succession, the full-color still picture docu- 
ment is decoded by the coding/decoding device 
1404 for dynamic picture image and still picture 
and written into the output frame memory 1405. As 

30 for the operation of the display composition 423 at 
this time, there is a method of displaying only the 
full-color still picture document without combining it 
with the personal computer signal in the same way 
as the case of dynamic picture image display. In 

35 order to comply with a request that the full-color 
still picture document should undergo correction 
and retouch by means of drawing and the position 
of explanation should be indicated by means of 
pointing, however, this information is stored in the 

40 display control section 415 of the meeting terminal 
main body 410 and superimposed over the full- 
color still picture document In the display composi- 
tion section. In case a full-color still picture is 
displayed in the picture 2 of Rg. 1, therefore, only 

45 the full-color image signal is displayed in the dy- 
namic picture image section, and the personal 
computer signal is so displayed in the full-color still 
picture image section as to be superimposed over 
the full-color image signal whereas only the per- 

50 sonal computer signal is displayed in the meeting 
progress menu display area 114. This switching is 
implemented by storing patterns associated with 
picture Nos. and display contents in the display 
composition section and specifying a pattern by 

55 means of the display changeover control signal 
supplied from the CPU 1403. The above described 
display changeover specification supplied from the 
CPU 1403 is executed upon receiving specification 
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supplied from the personal computer main body 
410 to the image CODEC 424. Fig. 15 shows an 
example of command and response specifications 
at this time. A command code corresponding to a 
display picture is supplied from the personal com- 
puter main body 410 to the image CODEC 424. In 
case of the full-color still picture document display, 
code data is also supplied from the personal com- 
puter main body 410 to the image CODEC 424. 
When processing is finished in the image CODEC, 
a response code is returned to the personal com- 
puter main body 410. 

Succeedingly, details of the FAX document 
display processing will now be described. Fig. 16 
shows the entire configuration regarding the FAX 
document processing. Numeral 1601 denotes a 
CPU of the peripheral controller, 1602 an l/F for the 
scanner 306, 1603 an l/F for the printer 307, 1604 
an image memory, and 1605 an image processing 
section for performing input/output control of the 
scanner and printer, coding/decoding of the FAX 
document and compression. Numeral 1606 de- 
notes a direct memory access (DM AC), 1607 a 
buffer memory, 1608 a two-way FIFO, and 1609 a 
DMAC. The scanner 306 and the printer 307 are 
the same as those of Fig. 3. The CPU (personal 
computer) 411, the display control section 415, the 
hard disk 414, the extension bus l/F 417, the pe- 
ripheral controller 421 , and the display information 
storage section (image memory) 416 are the same 
as those of Fig. 4. As for the flow of FAX document 
processing as a whole, a document inputted from 
the scanner 306 undergoes coding processing in 
the peripheral controller 421 and stored in the hard 
disk 414. At the time of display, the document is 
read out from the hard disk 414, decoded in the 
peripheral controller 421, and thereafter transferred 
to the display information storage section 416. A 
portion to be displayed is transferred to the display 
control section 415. The display method will here- 
after be described in detail. If the FAX document 
display is specified in the picture 120 by selecting 
the page name selection menu 508 of Fig. 8 t code 
data of the specified document is searched in the 
hard disk 414 and taken into the display informa- 
tion storage section 416. Thereafter, the code data 
is decoded in the peripheral controller 421. At this 
time, notification of an event between the personal 
computer CPU 410 and the peripheral controller 
421 is performed by an interrupt. Independent op- 
eration of the personal computer CPU 411 and the 
peripheral controller CPU 1601 is thus realized. 
Further, in order to increase the data transfer rate, 
data transfer on the personal computer bus is ad- 
ministered by the exterior DMAC 1609 and data 
transfer on the system bus is administered by the 
DMAC 1606. At the time of decoding, the personal 
computer requests for decoding and the code data 



is transferred to the buffer memory 1607 in accor- 
dance with the above described method. Then the 
code data is transferred to the input/output control 
and image processing section 1605 by the DMAC 

5 1606 and decoded therein. The result of decoding 
is developed into the image memory 1604, and the 
personal computer is informed of the end of pro- 
cessing. Succeedingly, full-scale transfer or com- 
pression transfer is requested by the personal 

w computer. Full-scale data or compressed data is 
read onto the system bus by the input/output con- 
trol and image processing section 1605 and stored 
into the buffer memory 1607 by the DMAC 1606. 
When storage of data into the buffer memory 1607 

75 has been finished, transferable communication in- 
terrupt is performed with respect to the personal 
computer. Thereafter, data are read out onto the 
personal computer bus via the FIFO by using the 
DMAC 1606. Upon receiving the transferable com- 

20 munication interrupt, the personal computer sets 
the quantity of transferred data into the DMAC 
1609 and starts it. As a result, data is transferred to 
the display information storage section 416. Finally, 
a portion to be displayed is transferred to the 

25 display control section 415. 

Fig. 17 shows an. example of configuration of 
the input/output control and image processing sec- 
tion 1605. 

Each of numerals 1701 and 1703 denote a 

30 DICEP-A (document image compression and ex- 
pansion processor-advanced). Numeral 1702 de- 
notes an ACRTC (advanced CRT controller). Nu- 
meral 1704 denotes a compression circuit. The 
DICEP-A 1701 is an LSI exclusively for coding and 

35 decoding the MMR (modified modified read) code, 
which complies with the international standards 
coding scheme of FAX documents, available from 
Hitachi Ltd. and executes data transfer on the im- 
age bus at the same time. The ACRTC 1702 is an 

40 LSI exclusively for displaying graphics and char- 
acters on a scanning CRT available from Hitachi 
Ltd.. The compression circuit 1704 is a circuit 
exclusively for executing compression of a FAX 
image with a ratio of 1/2 in the longitudinal and 

45 lateral directions. As for the processing contents of 
respective sections, the DICEP-A 1701 performs 
printer output control of the image memory, and 
the ACRTC 1702 performs development process- 
ing upon drawing data transmitted from the per- 

50 sonal computer at the time of printer output. The 
DICEP-A 1703 controls inputting from the scanner 
306 to the image memory 1604, exercises MMR 
decoding over data of the image memory 1604 to 
output it to the system bus, and exercises MMR 

55 decoding over MMR code data transmitted from 
the personal computer 410 to develop it into the 
image memory 1604. Further, when data of the 
image memory 1604 is to be transmitted as it is to 
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the personal computer 401, the DICEP-A 1703 
takes data out and outputs it to the system bus. 
The compression circuit 1704 compresses data of 
the image memory 1604 and outputs the result to 
the system bus. 

The display method of a FAX document on the 
display will now be described. In order to deal with 
an A4 full-size document in the terminal, 2376 x 
1728 pixels are demanded. A display having typical 
dimensions in both longitudinal and lateral direc- 
tions cannot provide a sufficient number of pixels. 
Unless a special display is used, the entire sheet of 
the document cannot be displayed. In case of 
partial display, scroll in the horizontal direction be- 
comes necessary, resulting in significant inconve- 
nience. In case of compression display, permis- 
sible image quality for a document cannot be main- 
tained unless the compression ratio is limited to 
approximately 1/2. As an embodiment, the case 
where a high resolution display (750 x 1120) in a 
typical personal computer is used will now be 
described. If an A4 full-size document is com- 
pressed to half in the longitudinal and lateral direc- 
tions, 1188 x 864 pixels are required and hence 
full-size display in the lateral direction becomes 
possible. Usually, therefore, images compressed to 
half in the longitudinal and lateral directions are 
displayed, and full-size images are displayed as 
occasion demands. At this time, full-size data is as 
large as approximately 500 KB in data quantity, 
resulting in long transfer time to the personal com- 
puter. Even if the full-size data is transferred, a 
larger part thereof is not displayed. In an embodi- 
ment, therefore, the full-size data is stored in the 
image memory of the peripheral controller 421, and 
a portion thereof to be displayed is transferred to 
the personal computer side as occasion demands 
and stored into the display information storage 
section 416 In the personal computer. 

The reason why such configuration is adopted - 
will now be described. Firstly, the transfer rate is 
approximately 125 KB and hence the transfer load 
is not so heavy. Secondly, the above described 
configuration lowers frequency in use of the exten- 
sion bus of the personal computer, which is used 
to implement high-speed scroll in the longitudinal 
direction in a race with other additional hardwares 
such as the image CODEC 424 and the commu- 
nication control section 425. 

Exchange of communication data in switching 
of meeting pictures will now be described in detail. 
Fig. 18 shows an example of its communication 
sequence. Data exchanged between the transmit- 
ting side (B) and the receiving side (C) are shown 
in area (A). Display pictures and user's action are 
shown in the transmitting side (B). Display pictures 
are shown in the receiving side (C). in the begin- 
ning, local pictures are displayed on both sides. 



When the meeting execution menu is selected at 
the terminal on one side, a line connection request 
for connecting lower layers of the OSI is issued. 
The receiving side returns an answer. The suc- 

s ceeding communication start request is a request 
for connection of a session layer of the OSI, and 
the communication start answer is an answer re- 
presenting the session layer connection. Connec- 
tion is thus performed up to the session layer, and 

10 the picture before meeting represented by picture 
1 of Fig. 1 is displayed on both sides. When the 
meeting suspension menu is selected in the picture 
before meeting, a communication suspension re- 
quest is issued as a request for opening the ses- 

15 sion layer, and a communication suspension an- 
swer is returned as the answer thereto. A line 
opening request is then issued as a request for 
opening the lower layer, and a line opening answer 
is returned in response thereto. The line is opened 

20 thereby, and pictures of both sides automatically 
return to local pictures. When the meeting start 
menu is selected in the picture before meeting, a 
meeting start request equivalent to the session 
activity start request is issued. A material prepared 

25 beforehand is automatically displayed on both 
screens. That is to say, a materia! presenting and 
explaining picture represented by the picture 2 of 
Fig. 1 is automatically displayed. Explanation is 
advanced by using menus located at the bottom of 

30 the picture 2. When the explanation is finished, a 
meeting end request corresponding to the session 
activity end request is issued. Upon receiving con- 
firmation of session activity end, the meeting after 
meeting represented by the picture 3 of Fig. 1 is 

35 automatically displayed on both screens. When the 
meeting restart menu is selected in the picture 
after meeting, a meeting restart request corre- 
sponding to the session activity resume request is 
issued and principal pictures of meeting are re- 

40 stored in both screens. When the communication 
end menu is selected in the picture after meeting, 
a communication end request including the session 
release request is issued. In response thereto, a 
communication end answer is returned. In succes- 

45 sion, the line opening request, which is a request to 
release lower layers, is issued. In response thereto, 
the line opening answer is returned, and the line is 
opened. Local pictures are automatically restored 
in both screens. 

so Fig. 19 shows an example of switching of pic- 
tures at the transmitting side. Displayed pictures 
and switching of them are shown. Numeral 1901 
denotes a local picture, and numeral 1902 denotes 
a display picture switching condition in the local 

55 picture 1901. Numeral 1903 denotes a picture be- 
fore meeting, and numeral 1904 denotes a display 
picture switching condition in the picture 1903 be- 
fore meeting. Numeral 1905 denotes a principal 

12 
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picture of the meeting, and numeral 1906 denotes 
a display picture switching condition in the princi- 
pal picture of the meeting. Numeral 1907 denotes a 
picture after meeting, and numeral 1908 denotes a 
display picture switching condition in the picture 
after meeting. Numeral 1909 denotes a local pic- 
ture. 

Fig. 20 shows an example of switching of pic- 
tures at the receiving side. Displayed pictures and 
switching of them are shown. Numeral 2002 de- 
notes a picture switching condition in the local 
picture. A closed switching arrow indicates that wait 
state is kept until meeting execution is received. 
Numeral 2004 denotes a picture switching con- 
dition in the picture before meeting. Numeral 2006 
denotes a picture switching condition in the princi- 
pal picture of meeting. Numeral 2008 denotes a 
picture switching condition in the picture after 
meeting. 

Switching of pictures are illustrated separately 
for the transmitting side and the receiving side. In 
an arbitrary picture, however, the terminal may 
become either of the transmitting side and the 
receiving side. 

In such a communication meeting that an iden- 
tical document is displayed on both displays (i.e., 
matching is taken) and explanations are given, ma- 
terials of both sides must be so matched before 
the start of explanations that documents are 
uniquely specified by identifiers. Such a method 
that this matching is taken in the material present- 
ing and explaining picture is conceivable. In order 
to allow concentration of mind upon explanations in 
meeting, materials used in the meeting are herein 
matched each other in the picture before meeting. 
Preferably, materials are automatically matched 
while the user is greeting by dynamic picture im- 
ages and voices. 

In case minutes are to be produced in the 
picture after meeting, the minutes may be auto- 
matically produced by the terminal side in accor- 
dance with a format prepared beforehand by the 
terminal side or the minutes may be produced in 
accordance with user's input. Minutes will be de- 
scribed later by referring to Fig. 24. 

In the embodiment shown in Fig. 1, switching 
between meeting pictures is automatically per- 
formed by menu selection. However, the present 
invention is not limited to this. Such a method that 
meeting pictures are automatically switched in ac- 
cordance with preset time may be adopted. At this 
time, provision of function of informing the user that 
the preset time has elapsed is a convenience to 
the user. Further, by provision of function of allow- 
ing the user to extend the preset time, use which is 
not limited by the preset time becomes possible. 

In the above described embodiment, common 
pictures are used throughout the meeting. How- 



ever, there is many-sided use allowing local work- 
ing during the meeting by configuration including 
both a window having display contents changed 
over in accordance with the progress of the meet- 

5 ing and a window having display contents changed 
locally irrespective of the progress of the meeting. 
By so configuring the system at this time that the 
window having display contents changed in accor- 
dance with the progress of the meeting can be 

70 manipulated by a directive on the window having 
display contents changed locally, slippage in dis- 
play synchronization of common image display pic- 
tures can be prevented. Further, by automatically, 
displaying the local window having display contents 

75 which are not changed with the progress of the 
meeting in response to a directive given on the 
window having display contents changed in accor- 
dance with the progress of the meeting, it becomes 
possible to make processing having a large amount 

20 of contents local working and thereby prevent slip- 
page in display synchronization. 

The present embodiment relates to a point to 
point communication meeting. In case communica- 
tion meeting among three or more points is allowed 

25 and participation in the course of the meeting is 
allowed, configuration having at least a principal 
picture of meeting and a picture for displaying 
greeting of a person newly participating in the 
course of meeting (i.e., a picture for identifying a 

30 person participating in the course of meeting) is 
effective in keeping a secret as described later. By 
displaying at this time at least a dynamic picture 
image together with voices in the picture for dis- 
playing greeting of a person newly participating in 

35 the course of meeting and displaying at least a 
medium other than a dynamic picture image to- 
gether with voices in the principal picture of meet- 
ing, it becomes possible to surely identify the other 
party. Further, by performing document matching 

40 during the display of the picture for displaying 
greeting of a person newly participating in the 
course of meeting, natural meeting participation 
can be realized. Further, in case succession in the 
course of the meeting is allowed, configuration 

45 having at least a principal picture of meeting and a 
picture for displaying greeting of a person seceding 
in the course of the meeting is effective in realizing 
a natural meeting environment. By displaying at 
this time at least a dynamic picture image together 

50 with voices in the picture for displaying greeting of 
a person seceding in the course of the meeting 
and displaying a medium other than a dynamic 
picture image together with voices in the principal 
picture of meeting, it becomes possible to provide 

55 an environment similar to an ordinary interview 
meeting. 

In the present embodiment, the meeting 
progress pattern is prepared beforehand at the 
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terminal side. It is effective that the user has a 
capability of registering the meeting process pat- 
tern so that setting complying with needs may be 
performed. Further, it is effective at this time to 
implement such configuration that an arbitrary 
scene using multiple media can be set, registered 
and operated. 

Further, in the present embodiment, terminals 
of both sides are on the same footing. By so 
configuring a communication meeting among two 
or more points that the right of communication may 
be moved by a terminal having leadership of com- 
munication as described later, it becomes possible 
to lighten the load of the terminal and provide 
environment similar to an ordinary meeting having 
chairman. 

A person so defined before the start of a meet- 
ing as to participate in it is automatically allowed to 
participate in the meeting at the person's wish. In 
case a person is newly called out to participate in 
the meeting as occasion demands, the person thus 
called out is identified and thereafter the person 
participates in the meeting. In case communication 
among multiple points is realized, it is effective in 
keeping a secret to thus distinguish the situation of 
the participant. 

Fig. 21 shows an example of display picture 
obtained when a meeting among two or more 
points is held. Numeral 2103 denotes a participant 
dynamic picture image display area. Numeral 2104 
denotes a participant dynamic picture image. As 
compared with the principal picture 120 of Rg. 1, 
the portion for displaying person dynamic picture 
images has been changed to comply with multiple 
points. As for the midway participation method 
during execution of the meeting, the following two 
methods are possible. The first method corre- 
sponds to the case where a person so defined 
beforehand as to participate in the meeting partici- 
pate in the meeting part-way. The second method 
corresponds to the case where a person is urged 
to participate in the meeting part-way and tell opin- 
ion as occasion demands. 

In the case a person so defined beforehand as 
to participate in the meeting participate in it part- 
way, the above described first method automati- 
cally matches meeting materials and allows auto- 
matic switching to the principal picture of meeting 
as shown in Rg. 23 at the person's wish. 

In case a person is urged to participate in the 
meeting part-way in accordance with the second 
method, suddenly switching to the principal picture 
of the meeting as in the first method poses a 
problem in keeping a secret because the person to 
be called out is not necessarily on the screen. 

Rg. 22 shows a picture for identifying a mid- 
way meeting participant. Numeral 2201 denotes a 
dynamic picture image display area for identifying 



midway participant. Numeral 2202 denotes a dy- 
namic picture image for identifying midway partici- 
pant. Numeral 2204 denotes a participation consent 
menu. Numeral 2205 denotes a participation cancel 

e menu. When it becomes necessary to identify the 
other party of the meeting in the course of the 
meeting, the other party is called out, and switch- 
ing to this picture is automatically performed when 
lines are connected. In case the other party is 

10 identified in this switched picture by means of the 
midway participant identifying dynamic picture im- 
age 2202 and voices, the participation consent 
menu 2204 is selected and switching to the picture 
of Rg. 23 is automatically performed. On the other 

76 hand, in case the other party is absent, the partici- 
pation cancel menu is selected. The line is auto- 
matically released and, the picture of Fig. 21 is 
restored. 

Further, in some cases of communication 

20 among multiple points, some of the participants do 
not take part, but desire to merely watch the meet- 
ing. In such a case, it is desirable that the commu- 
nication network 202 or the electronic exchange 
203 administer separately telemeeting terminals 

25 taking part is meeting progress and capable of 
switching display pictures and telemeeting termi- 
nals which do not take part in meeting progress 
and which share the picture. In not only this case 
but also cases where communication in performed 

so among a large number of points, connecting termi- 
nal devices in a ringlike network causes degraded 
real time property and increase in load of telemeet- 
ing terminals. In these cases, therefore, it is desir- 
able that the electronic exchange 203 or the com- 

35 munication network 202 has additional function for 
exercising centralized control and starlike network 
configuration is used. 

With reference to Fig. 6, selecting the page 
name selection menu 508 or the like changes the 

40 display of the common image display area 111. and 
switches pictures. It is conceivable to provide 
means for preserving minutes in the picture at the 
time of switching. Rg. 24 shows an example there- 
of. Numeral 2401 denotes a document, 2402 page 

45 1. 2403 page n, 2404 block 1, 2405 block 2, 2406 
text unit 1, 2407 text unit 2, and 2408 text unit 3. 
Each material typically has a hierarchical structure 
ranging from document to page. In the initial state, 
page 1 in located under the document 2401 (pages 

so 2 to n are not considered) and the text unit 1 which 
is actual code information of the document is lo- 
cated under page 1. When the displayed page is 
changed over, one's own drawing data and drawing 
date of the other party on that page are preserved 

55 as minutes. To be concrete, drawing data actually 
drawn by a person oneself and drawing data ac- 
tually drawn by the other party are so preserved 
respectively under the block 1 Indicating that it is 
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one's own drawing data and under the block 2 
indicating drawing data of the other party as to be 
associated with page 1 . This data preservation form 
is based upon layout structures in ODA (office 
document architecture) are ODIF (office document 
interchange format). 

The token ownership display area 504 was 
described before with reference to Fig. 6. It is 
effective in strengthening a sense of security to 
display whether the right of token is present or not, 
which represents whether the right of token is 
present or not, on a part of the screen. As for 
transfer of the right of communication, it is conceiv- 
able to transfer the right of communication by 
selecting a token right display area defined as 
menu. However, it is inconvenient to the user to 
make the user conscious of the right of token 
which is the concept peculiar to the communication 
technique. Therefore, it is desirable that the right of 
token is moved when the user takes action needing 
the right of token (such as alteration of display 
page). In the above described embodiment, the 
right of token is needed for every manipulation in 
progress of meeting. For specific media commu- 
nication such as drawing and pointing, material 
transfer communication, and the like, however, it is 
made possible to freely transfer data without need- 
ing the right of token. As for drawing, for example, 
an environment allowing mutual, free drawing like 
an ordinary interview reference is created on the 
communication media as well. Further, time-con- 
suming meeting advance manipulation such as 
"material transfer" is executed in parallel to an 
ordinary job for executing a communication meet- 
ing as a sub-job different from the ordinary job. In 
this way, token ownership is provided for each 
media unit or each classification of meeting ad- 
vance manipulation, and parallel processing is per- 
formed, an environment for smooth meeting ad- 
vance manipulation being thus realized. 

In the example heretofore described, menu se- 
lection is used as means for informing the terminal 
that the other party has been identified. However, it 
is also conceivable that the terminal device side 
automatically identifies the other party by means of 
speech recognition and/or image recognition in or- 
der to further lighten the burden of the user. 

The present invention brings about the follow- 
ing effects. 

® Since switching to, a picture for presenting 
and explaining a material (i.e., principal picture of 
meeting) can be made after identifying the other 
party of the meeting, it is possible to keep the 
materia! secret 

@ Since the other party intended can be 
identified in a picture for identifying the other party 
of the meeting (i.e.. picture before meeting), mis- 
identification can be reduced. 



©Since a display picture has multiple windows 
and hence various media can be selected and 
combined for display, it is possible to provide an 
environment convenient to the user. 

5 @ Since the display picture has fixed multiple 

windows depending upon the progress of meeting, 
it is possible to provide such a media manipulation 
environment that the user is not perplexed about 
how the user manipulates windows. 

10 ® Since the picture before meeting is auto- 
matically, displayed when communication lines are 
connected, the user can be smoothly carried on 
the flow of the meeting prepared at the terminal 
side beforehand. 

T5 © Since media other than a dynamic picture 
image can be displayed in the picture before meet- 
ing for identifying the other party in conjunction 
with voices, the other party can be identified by the 
sense of sight and the sense of hearing in the 

20 picture before meeting. 

© By using media other than a dynamic 
picture image in the principal picture of meeting, 
diverse presentation becomes possible. 

(§) Since menu selection can be made as 

25 means for informing the terminal that the other 
party has been identified, the manipulation is sim- 
ple. 

(D Since the principal picture of meeting has 
both an area for displaying an image common to 
30 the terminal of the other party and a menu display 
area for controlling the common image, smooth 
meeting operation becomes possible. 

© 

35 

Since a menu for informing the terminal that the 
other party has been identified is displayed in a 
common display area independently of a meeting 
40 support menu displayed in the menu display area, 
it is possible to provide a easily selectable, conve- 
nient environment. 

45 

In the picture for identifying the other party before 
the meeting, the image inputted at the terminal of 
the other party is displayed at its own terminal. 
Thereby, mutual identification can be performed. In 

60 the principal picture of meeting, the material of its 
own terminal or the terminal of the other party is 
displayed on both its own terminal and the terminal 
of the other party. During the meeting, therefore, it 
is possible to advance the meeting while watching 

55 the same picture. 
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1. A telemeeting terminal device for performing 
communication between remote locations by using 
multiple media, said telemeeting terminal device 
comprising: 

means for identifying the other party of a meeting . 

2. The telemeeting terminal device according to 
Claim 1, wherein a medium for identifying the other 
party of meeting comprises an image display de- 
vice. 

3. The telemeeting terminal device according to 
Claim 2, wherein images displayed on the image 
display device comprises multiwindow display im- 
ages., 

4. The telemeeting terminal device according to 
Claim 3, wherein positions of multiple windows are 
fixed on a display picture. 

5. The telemeeting terminal device according to 
Claim 1, wherein the means for identifying the 
other party of the meeting comprises: 

means for displaying an image transmitted from 
the other party; and 

means for inputting result of identifying the other 
party intended on the basis of the resulted image. 

6. The telemeeting terminal device according to 
Claim 5, wherein the means for inputting result of 
identifying the other party intended comprises 
means for selecting a menu displayed on one's 
own terminal device side. 

7. The telemeeting terminal device according to 
Claim 1, wherein a medium for identifying the other 
party comprises a speech recognition device. 

8. The telemeeting terminal device according to 
Claim 1, wherein the means for identifying the 
other party of the meeting comprises means for 
recognizing a recognition media peculiar to the 
other party. 

9. The telemeeting terminal device according to 
Claim 1, wherein the means for identifying the 
other party of the meeting comprises means for 
selecting a menu on a picture displayed on one's 
own terminal device side. 

10. The telemeeting terminal device in accordance 
with one of the preceding claims, wherein said 
means for identifying the other party are adapted to 
identify said other party before the meeting. 

11. The telemeeting terminal device according to 
one of the preceding claims, further comprising: 
means for presenting and explaining a material; 
and 

switching means for switching a state from said 
identifying means to said material presenting 
means. 

12. The telemeeting terminal device according to 
claim 11, wherein said switching means are adapt- 
ed for switching from an identifying picture before 
meeting to a principal picture for presenting and 
explaining materials. 



13. The telemeeting terminal device according to 
Claim 12, wherein switching from the picture before 
meeting to the principal picture is executed after 
the other party intended has been identified. 

5 14. The telemeeting terminal device according to 
Claim 12, further comprising means responsive to 
identification of the other party of a meeting for 
matching a material for the meeting displayed on 
the terminal device of the other party with that 

w displayed on one's own terminal device. 

15. The telemeeting terminal device according to 
Claim 12, further comprising means responsive to 
identification of the other party of the meeting for 
automatically executing the matching operation 

is without a directive of an operator. 

16. The telemeeting terminal device according to 
Claim 12, wherein the picture before meeting and 
the principal picture are displayed with multiple 
windows. 

20 17. The telemeeting terminal device according to 
Claim 12, wherein positions of multiple windows 
are fixed on a display picture. 

18. The telemeeting terminal device according to 
Claim 12, further comprising means responsive to 

25 connection of communication lines for automati- 
cally displaying the picture before meeting. 

19. The telemeeting terminal device according to 
Claim 12, wherein the picture before meeting com- 
prises a selection menu for inputting the result of 

30 identification of the other party intended. 

20. The telemeeting terminal device according to 
Claim 12, wherein an image display area of the 
principal picture comprises: 

an area for displaying an image common to the 
35 terminal device of the other party and one's own 
terminal device; and 

a menu display area for controlling said common 
image display area and displaying a meeting sup- 
port menu to support the advance of a meeting. 

40 21. The telemeeting terminal device according to 
Claim 20, wherein a menu for informing one's own 
terminal device that the other party has been iden- 
tified is displayed on the common image display 
area provided independently of said meeting sup- 

45 port menu. 

22. The telemeeting terminal device according to 
Claim 12, wherein at least a dynamic picture image 
medium is displayed on the picture before meeting 
and at least a medium different from said dynamic 

so picture image medium is displayed on the principal 
picture. 

23. The telemeeting terminal device according to 
Claim 12, further comprising: 

a first window having display contents changed 
55 according to the progress of a meeting; and 

a second window having display contents changed 
irrespectively of the progress of the meeting. 

24. The telemeeting terminal device according to 
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Claim 12, further comprising means for executing 
at least one of actions for setting, registering and 
operating the picture before meeting and the princi- 
pal picture. 

25. The telemeeting terminal device according to 
Claim 23, further comprising means for switching to 
said first window upon a directive given on said 
second window. 

26. The telemeeting terminal device according to 
Claim 23, further comprising means for switching to 
said second window upon a directive given on said 
first window. 

27. The telemeeting terminal device according to 
one of the preceding claims, wherein executing 
means are provided for executing on the basis of 
preset time, at least one of switching from a picture 
before meeting for identifying the other party to a 
principal picture for presenting and explaining ma- 
terials and switching to a picture after meeting for a 
participant's identification of end of a meeting. 

28. The telemeeting terminal device according to 
Claim 27, further comprising means for outputting 
elapse of the preset time. 

29. The telemeeting terminal device according to 
Claim 27, further comprising means for changing 
the preset time. 

30. The telemeeting terminal device according to 
one of the preceding claims, further comprising: 
means for releasing communication lines on the 
basis of result of identification of the other party in 
a picture before meeting for identifying the other 
party. 

31. The telemeeting terminal device according to 
one of the preceding claims, further comprising: 
means for displaying an image representing 
presence/absence of a token ownership on said 
principal picture; and 

means for allowing a participant manipulating a 
terminal device provided with said token ownership 
to switch a common display area of said principal 
picture. 

32. The telemeeting terminal device according to 
Claim 31 , further comprising: 

means responsive to action performed by operator 
manipulating said terminal device to change con- 
tents of a display picture displayed on said termi- 
nal device for executing movement of token owner- 
ship between terminal devices. 

33. The telemeeting terminal device according to 
Claim 31 or 32, further comprising: 

means for executing communication of a predeter- 
mined medium without obtaining token ownership 
existing between terminal devices. 

34. The telemeeting terminal device according to 
one of the claims 31 to 33, further comprising: 
means for giving token ownership existing between 
terminal devices to each of kinds of at least either 
of communication media and meeting advancing 



manipulation. 

35. A manipulation method of telemeeting terminal 
device for performing communication betveen re- 
mote locations by using multiple media, said ma- 

5 nipulation method comprising the step of: 
identifying the other party of a meeting. 

36. The manipulation method of telemeeting termi- 
nal device according to Claim 35, wherein said 
step of identifying the other party of the meeting 

70 comprises the steps of: 

displaying an image transmitted from the other 
party, and 

inputting the result of identifying the other party 
intended, on the basis of said displayed image. 

75 37. The manipulation method of telemeeting termi- 
nal device according to Claim 35, wherein said 
step of identifying the other party of the meeting 
comprises the step of inputting the result of rec- 
ognition of voices transmitted from the other party. 

20 38. The manipulation method according to one of 
the Claims 35 to 37, wherein the identification step 
is carried out before entering the meeting. 

39. The manipulation method according to one of 
the Claims 35 to 38, wherein said step of identify- 

25 ing the other party of the meeting comprises the 
steps of: 

inputting a recognition medium peculiar to the oth- 
er party; 

registering recognition media of meeting partici- 
30 pants; 

collating said inputted recognition medium with 
said registered recognition media; and outputting 
the result of said collation. 

40. A telemeeting communication system for per- 
35 forming communication among a plurality of 

telemeeting terminal devices connected via at least 
one of an electronic exchange and a communica- 
tion network and located at remote locations by 
using multiple media, said telemeeting terminal 

40 system comprising: 

means for allowing said electronic exchange and 
communication network to administer telemeeting 
terminal devices displaying pictures on the basis of 
progress of a meeting in distinction from telemeet- 

45 ing terminal devices displaying pictures irrespec- 
tively of the progress of the meeting. 
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